Batch access (ex: gliders)

Packages used

To use python to access the ERDDAP server directly from your python script or jupyter-notebook, you will need

• ERDDAPY

• Xarray

• netcdf4

• matplotlib

• folium

• numpy

Note

The package netcdf4 develop by UNIDATA is not needed in the import part of the python script. However, it is the essential package that support netCDF format output from Xarray. The package matplotlib is also not needed in the import part of the python script. It is the essential package that support quick visualization from Xarray.

In this page, we demonstrate how to extract/download data directly from a ERDDAP server and perform data processing, visualization, and export data in python environment.

Tip

Understanding of the ERDDAP server and what it provides is highly recommended before reading the following intructions.

Import python packages

import xarray as xr
from erddapy import ERDDAP

• xarray is used for data processing and netCDF file output.

• erddapy is used to access the ERDDAP server.

Both package-webpages have more detail explanation on its full range of functionalities. Here we will mainly focusing on getting the data to be displayer and downloaded.

Access IOOS Glider DAC (TableDAP) data

In this demostration, we will be getting the glider data from IOOS Glider Data Archived Center

Firstly, the way to use the erddapy is to setup the destination ERDDAP server as an object in python through ERDDAP (a python class)

#### access the ERDDAP server 
e = ERDDAP(
    server="https://gliders.ioos.us/erddap/",             # The URL that the ERDDAP server has
    protocol="tabledap",                                  # The data type (griddap or tabledap)
    response="opendap",                                   # different output data type that provided by ERDDAP server       
)

By executing the above code block, we have already setup the connection with the desired ERDDAP server. To request a the dataset list on the server, we need to set the dataset_id = allDatasets(e.g. https://gliders.ioos.us/erddap/tabledap/allDatasets.html).

To set the dataset_id, execute

# set the dataset id name 
e.dataset_id = "allDatasets"

Download data list

Now, all the setting for downloading the dataset list is complete. By executing the following, we convert the list into a pandas dataframe object that contain all the dataset info on the ERDDAP server

df = e.to_pandas()

Let’s take a peak at the first five glider deployments

df.head(5)

	datasetID	accessible	institution	dataStructure	cdm_data_type	class	title	minLongitude (degrees_east)	maxLongitude (degrees_east)	longitudeSpacing (degrees_east)	...	fgdc	iso19115	metadata	sourceUrl	infoUrl	rss	email	testOutOfDate	outOfDate	summary
0	allDatasets	public	IOOS Glider DAC	table	Other	EDDTableFromAllDatasets	* The List of All Active Datasets in this ERDD...	NaN	NaN	NaN	...	NaN	NaN	https://gliders.ioos.us/erddap/info/allDataset...	https://gliders.ioos.us/erddap	https://gliders.ioos.us/erddap	https://gliders.ioos.us/erddap/rss/allDatasets...	https://gliders.ioos.us/erddap/subscriptions/a...	NaN	NaN	This dataset is a table which has a row of inf...
1	amelia-20180501T0000	public	Virginia Institute of Marine Science - William...	table	TrajectoryProfile	EDDTableFromNcFiles	amelia-20180501T0000	-75.079579	-74.445788	NaN	...	https://gliders.ioos.us/erddap/metadata/fgdc/x...	https://gliders.ioos.us/erddap/metadata/iso191...	https://gliders.ioos.us/erddap/info/amelia-201...	(local files)	https://gliders.ioos.us/erddap/	https://gliders.ioos.us/erddap/rss/amelia-2018...	https://gliders.ioos.us/erddap/subscriptions/a...	NaN	NaN	This is a test mission for the Northwest Passa...
2	amelia-20200825T1929	public	Virginia Institute of Marine Science - William...	table	TrajectoryProfile	EDDTableFromNcFiles	amelia-20200825T1929	-75.360315	-74.443807	NaN	...	https://gliders.ioos.us/erddap/metadata/fgdc/x...	https://gliders.ioos.us/erddap/metadata/iso191...	https://gliders.ioos.us/erddap/info/amelia-202...	(local files)	https://gliders.ioos.us/erddap/	https://gliders.ioos.us/erddap/rss/amelia-2020...	https://gliders.ioos.us/erddap/subscriptions/a...	NaN	NaN	This project supports the deployment and realt...
3	amelia-20201015T1436	public	Virginia Institute of Marine Science - William...	table	TrajectoryProfile	EDDTableFromNcFiles	amelia-20201015T1436	-74.947858	-74.405013	NaN	...	https://gliders.ioos.us/erddap/metadata/fgdc/x...	https://gliders.ioos.us/erddap/metadata/iso191...	https://gliders.ioos.us/erddap/info/amelia-202...	(local files)	https://gliders.ioos.us/erddap/	https://gliders.ioos.us/erddap/rss/amelia-2020...	https://gliders.ioos.us/erddap/subscriptions/a...	NaN	NaN	This project supports the deployment and realt...
4	amlr01-20181216T0641-delayed	public	NOAA SWFSC Antarctic Ecosystem Research Division	table	TrajectoryProfile	EDDTableFromNcFiles	amlr01-20181216T0641-delayed	-61.756371	-56.996472	NaN	...	https://gliders.ioos.us/erddap/metadata/fgdc/x...	https://gliders.ioos.us/erddap/metadata/iso191...	https://gliders.ioos.us/erddap/info/amlr01-201...	(local files)	https://gliders.ioos.us/erddap/	https://gliders.ioos.us/erddap/rss/amlr01-2018...	https://gliders.ioos.us/erddap/subscriptions/a...	NaN	NaN	These data are part of the U.S. Antarctic Mari...

5 rows × 36 columns

Now, let us create a list of dataset_id. This helps if user want to download all datasets on the server.

dataset_ids = []
for dataset_id in df['datasetID']:
    if dataset_id != 'allDatasets':
        dataset_ids.append(dataset_id)

The first five dataset_id is shown below.

dataset_ids[:5]

['amelia-20180501T0000',
 'amelia-20200825T1929',
 'amelia-20201015T1436',
 'amlr01-20181216T0641-delayed',
 'amlr01-20191206T0452-delayed']

Picking one dataset to investigate on the fly

# e.dataset_id = 'ce_311-20190703T1802-delayed'    # feel free to uncomment and test
# e.dataset_id = 'amelia-20201015T1436'            
e.dataset_id = 'UW685-20230125T0000'             # feel free to uncomment and test
df = e.to_pandas()
ds = df.to_xarray()

Note

Here, we try to extract the data from the server by first converting it into a Pandas dataframe then into a Xarray dataset. A direct e.to_xarray() would result in another data structure which is harder to visualize in this notebook.

Converting from the dataframe to the xarray helps us to convert into a gridded dataset in the later on preprocessing step. But if that is not needed, ds = df.to_xarray() is not a necessary step to extract the dataset from the server.

Preprocessing the data

To focus on the vertical profile, we first remove row that does not contain the depth value.

ds_transect = ds.where(ds['depth (m)'].notnull(),drop=True)   # only preserve the profile related data

The original dataset structure is

ds

<xarray.Dataset>
Dimensions:                      (index: 820119)
Coordinates:
  * index                        (index) int64 0 1 2 3 ... 820116 820117 820118
Data variables: (12/49)
    trajectory                   (index) object 'UW685-20230125T0000' ... 'UW...
    wmo_id                       (index) int64 4803976 4803976 ... 4803976
    profile_id                   (index) int64 1 1 1 1 1 ... 1008 1008 1008 1008
    time (UTC)                   (index) object '2023-01-26T20:10:27Z' ... '2...
    latitude (degrees_north)     (index) float64 40.83 40.83 ... 41.05 41.05
    longitude (degrees_east)     (index) float64 -124.4 -124.4 ... -127.1 -127.1
    ...                           ...
    time_uv (UTC)                (index) object '2023-01-26T20:10:37Z' ... '2...
    time_uv_qc                   (index) int64 1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1
    u (m s-1)                    (index) float64 -0.06249 -0.06249 ... nan nan
    u_qc                         (index) float64 3.0 3.0 3.0 3.0 ... nan nan nan
    v (m s-1)                    (index) float64 -0.00278 -0.00278 ... nan nan
    v_qc                         (index) float64 3.0 3.0 3.0 3.0 ... nan nan nan

xarray.Dataset

• Dimensions:
  • index: 820119
• Coordinates: (1)
  • index
    (index)
    int64
    0 1 2 3 ... 820116 820117 820118

    array([     0,      1,      2, ..., 820116, 820117, 820118])

• Data variables: (49)
  • trajectory
    (index)
    object
    'UW685-20230125T0000' ... 'UW685...

    array(['UW685-20230125T0000', 'UW685-20230125T0000',
           'UW685-20230125T0000', ..., 'UW685-20230125T0000',
           'UW685-20230125T0000', 'UW685-20230125T0000'], dtype=object)

  • wmo_id
    (index)
    int64
    4803976 4803976 ... 4803976 4803976

    array([4803976, 4803976, 4803976, ..., 4803976, 4803976, 4803976])

  • profile_id
    (index)
    int64
    1 1 1 1 1 ... 1008 1008 1008 1008

    array([   1,    1,    1, ..., 1008, 1008, 1008])

  • time (UTC)
    (index)
    object
    '2023-01-26T20:10:27Z' ... '2023...

    array(['2023-01-26T20:10:27Z', '2023-01-26T20:10:27Z',
           '2023-01-26T20:10:27Z', ..., '2023-05-23T10:49:13Z',
           '2023-05-23T10:49:13Z', '2023-05-23T10:49:13Z'], dtype=object)

  • latitude (degrees_north)
    (index)
    float64
    40.83 40.83 40.83 ... 41.05 41.05

    array([40.83496667, 40.83496667, 40.83496667, ..., 41.05235689,
           41.05235689, 41.05235689])

  • longitude (degrees_east)
    (index)
    float64
    -124.4 -124.4 ... -127.1 -127.1

    array([-124.44637307, -124.44637307, -124.44637307, ..., -127.12167483,
           -127.12167483, -127.12167483])

  • depth (m)
    (index)
    float64
    0.9907 0.9497 ... 0.5305 0.1026

    array([0.9907086 , 0.9496902 , 0.8640232 , ..., 1.1711848 , 0.53051794,
           0.1025867 ])

  • cdom (ppb/L)
    (index)
    float64
    1.348 1.348 1.348 ... 1.798 1.798

    array([1.3485, 1.3485, 1.3485, ..., 1.7081, 1.798 , 1.798 ])

  • cdom_qc
    (index)
    int64
    0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0

    array([0, 0, 0, ..., 0, 0, 0])

  • conductivity (S m-1)
    (index)
    float64
    nan nan nan nan ... 3.759 nan nan

    array([      nan,       nan,       nan, ..., 3.7591789,       nan,
                 nan])

  • conductivity_qc
    (index)
    int64
    4 4 4 4 4 4 4 4 ... 1 1 1 1 1 1 4 4

    array([4, 4, 4, ..., 1, 4, 4])

  • density (kg m-3)
    (index)
    float64
    nan nan nan ... 1.025e+03 nan nan

    array([      nan,       nan,       nan, ..., 1024.5771,       nan,
                 nan])

  • density_qc
    (index)
    int64
    1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1 1

    array([1, 1, 1, ..., 1, 1, 1])

  • depth_qc
    (index)
    int64
    0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0

    array([0, 0, 0, ..., 0, 0, 0])

  • fluorescence (microgram/L)
    (index)
    float64
    0.4148 0.3416 ... 0.549 0.5246

    array([0.4148, 0.3416, 0.3538, ..., 0.5368, 0.549 , 0.5246])

  • fluorescence_qc
    (index)
    int64
    0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0

    array([0, 0, 0, ..., 0, 0, 0])

  • instrument_ctd (1)
    (index)
    float64
    nan nan nan nan ... nan nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • lat_qc
    (index)
    int64
    0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0

    array([0, 0, 0, ..., 0, 0, 0])

  • lat_uv (degrees_north)
    (index)
    float64
    40.83 40.83 40.83 ... 41.05 41.05

    array([40.83496667, 40.83496667, 40.83496667, ..., 41.05235689,
           41.05235689, 41.05235689])

  • lat_uv_qc
    (index)
    int64
    1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1 1

    array([1, 1, 1, ..., 1, 1, 1])

  • latitude_qc
    (index)
    int64
    1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1 1

    array([1, 1, 1, ..., 1, 1, 1])

  • lon_qc
    (index)
    int64
    0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0

    array([0, 0, 0, ..., 0, 0, 0])

  • lon_uv (degrees_east)
    (index)
    float64
    -124.4 -124.4 ... -127.1 -127.1

    array([-124.44637307, -124.44637307, -124.44637307, ..., -127.12167483,
           -127.12167483, -127.12167483])

  • lon_uv_qc
    (index)
    int64
    1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1 1

    array([1, 1, 1, ..., 1, 1, 1])

  • longitude_qc
    (index)
    int64
    1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1 1

    array([1, 1, 1, ..., 1, 1, 1])

  • opbs (m^-1)
    (index)
    float64
    0.001734 0.001614 ... 0.0002276

    array([0.0017343 , 0.00161437, 0.00163897, ..., 0.0001968 , 0.00020603,
           0.00022755])

  • opbs_qc
    (index)
    int64
    0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0

    array([0, 0, 0, ..., 0, 0, 0])

  • oxygen (micromol/kg)
    (index)
    float64
    nan nan nan nan ... nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • oxygen_qc
    (index)
    int64
    4 4 4 4 4 4 4 4 ... 1 1 1 1 1 3 4 4

    array([4, 4, 4, ..., 3, 4, 4])

  • platform_meta (1)
    (index)
    float64
    nan nan nan nan ... nan nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • precise_lat (degrees_north)
    (index)
    float64
    40.83 40.83 40.83 ... 41.05 41.05

    array([40.83468333, 40.83468327, 40.83468315, ..., 41.05268175,
           41.05268137, 41.05268001])

  • precise_lat_qc
    (index)
    float64
    nan nan nan nan ... nan nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • precise_lon (degrees_east)
    (index)
    float64
    -124.4 -124.4 ... -127.1 -127.1

    array([-124.4462    , -124.44620185, -124.44620555, ..., -127.12957832,
           -127.12958763, -127.12959082])

  • precise_lon_qc
    (index)
    float64
    nan nan nan nan ... nan nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • precise_time (UTC)
    (index)
    object
    '2023-01-26T20:06:39Z' ... '2023...

    array(['2023-01-26T20:06:39Z', '2023-01-26T20:06:44Z',
           '2023-01-26T20:06:49Z', ..., '2023-05-23T12:34:17Z',
           '2023-05-23T12:34:22Z', '2023-05-23T12:34:27Z'], dtype=object)

  • precise_time_qc
    (index)
    int64
    1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1 1

    array([1, 1, 1, ..., 1, 1, 1])

  • pressure (dbar)
    (index)
    float64
    0.9992 0.9578 ... 0.5347 0.1037

    array([0.99918216, 0.9578449 , 0.87150395, ..., 1.1801987 , 0.53466433,
           0.10365458])

  • pressure_qc
    (index)
    int64
    0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0

    array([0, 0, 0, ..., 0, 0, 0])

  • salinity (0.001)
    (index)
    float64
    nan nan nan nan ... nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • salinity_qc
    (index)
    int64
    4 4 4 4 4 4 4 4 ... 1 1 1 1 1 3 4 4

    array([4, 4, 4, ..., 3, 4, 4])

  • temperature (Celsius)
    (index)
    float64
    11.15 11.15 11.15 ... 12.23 12.19

    array([11.150965 , 11.150085 , 11.149957 , ..., 12.228412 , 12.2253275,
           12.190469 ])

  • temperature_qc
    (index)
    int64
    1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1 1

    array([1, 1, 1, ..., 1, 1, 1])

  • time_qc
    (index)
    int64
    1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1 1

    array([1, 1, 1, ..., 1, 1, 1])

  • time_uv (UTC)
    (index)
    object
    '2023-01-26T20:10:37Z' ... '2023...

    array(['2023-01-26T20:10:37Z', '2023-01-26T20:10:37Z',
           '2023-01-26T20:10:37Z', ..., '2023-05-23T10:49:13Z',
           '2023-05-23T10:49:13Z', '2023-05-23T10:49:13Z'], dtype=object)

  • time_uv_qc
    (index)
    int64
    1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1 1

    array([1, 1, 1, ..., 1, 1, 1])

  • u (m s-1)
    (index)
    float64
    -0.06249 -0.06249 ... nan nan

    array([-0.06249343, -0.06249343, -0.06249343, ...,         nan,
                   nan,         nan])

  • u_qc
    (index)
    float64
    3.0 3.0 3.0 3.0 ... nan nan nan nan

    array([ 3.,  3.,  3., ..., nan, nan, nan])

  • v (m s-1)
    (index)
    float64
    -0.00278 -0.00278 ... nan nan

    array([-0.0027802, -0.0027802, -0.0027802, ...,        nan,        nan,
                  nan])

  • v_qc
    (index)
    float64
    3.0 3.0 3.0 3.0 ... nan nan nan nan

    array([ 3.,  3.,  3., ..., nan, nan, nan])

• Indexes: (1)
  • index
    PandasIndex

    PandasIndex(RangeIndex(start=0, stop=820119, step=1, name='index'))

• Attributes: (0)

After the .where filtering, the dataset structure becomes

ds_transect

<xarray.Dataset>
Dimensions:                      (index: 820119)
Coordinates:
  * index                        (index) int64 0 1 2 3 ... 820116 820117 820118
Data variables: (12/49)
    trajectory                   (index) object 'UW685-20230125T0000' ... 'UW...
    wmo_id                       (index) float64 4.804e+06 ... 4.804e+06
    profile_id                   (index) float64 1.0 1.0 ... 1.008e+03 1.008e+03
    time (UTC)                   (index) object '2023-01-26T20:10:27Z' ... '2...
    latitude (degrees_north)     (index) float64 40.83 40.83 ... 41.05 41.05
    longitude (degrees_east)     (index) float64 -124.4 -124.4 ... -127.1 -127.1
    ...                           ...
    time_uv (UTC)                (index) object '2023-01-26T20:10:37Z' ... '2...
    time_uv_qc                   (index) float64 1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0
    u (m s-1)                    (index) float64 -0.06249 -0.06249 ... nan nan
    u_qc                         (index) float64 3.0 3.0 3.0 3.0 ... nan nan nan
    v (m s-1)                    (index) float64 -0.00278 -0.00278 ... nan nan
    v_qc                         (index) float64 3.0 3.0 3.0 3.0 ... nan nan nan

xarray.Dataset

• Dimensions:
  • index: 820119
• Coordinates: (1)
  • index
    (index)
    int64
    0 1 2 3 ... 820116 820117 820118

    array([     0,      1,      2, ..., 820116, 820117, 820118])

• Data variables: (49)
  • trajectory
    (index)
    object
    'UW685-20230125T0000' ... 'UW685...

    array(['UW685-20230125T0000', 'UW685-20230125T0000',
           'UW685-20230125T0000', ..., 'UW685-20230125T0000',
           'UW685-20230125T0000', 'UW685-20230125T0000'], dtype=object)

  • wmo_id
    (index)
    float64
    4.804e+06 4.804e+06 ... 4.804e+06

    array([4803976., 4803976., 4803976., ..., 4803976., 4803976., 4803976.])

  • profile_id
    (index)
    float64
    1.0 1.0 1.0 ... 1.008e+03 1.008e+03

    array([1.000e+00, 1.000e+00, 1.000e+00, ..., 1.008e+03, 1.008e+03,
           1.008e+03])

  • time (UTC)
    (index)
    object
    '2023-01-26T20:10:27Z' ... '2023...

    array(['2023-01-26T20:10:27Z', '2023-01-26T20:10:27Z',
           '2023-01-26T20:10:27Z', ..., '2023-05-23T10:49:13Z',
           '2023-05-23T10:49:13Z', '2023-05-23T10:49:13Z'], dtype=object)

  • latitude (degrees_north)
    (index)
    float64
    40.83 40.83 40.83 ... 41.05 41.05

    array([40.83496667, 40.83496667, 40.83496667, ..., 41.05235689,
           41.05235689, 41.05235689])

  • longitude (degrees_east)
    (index)
    float64
    -124.4 -124.4 ... -127.1 -127.1

    array([-124.44637307, -124.44637307, -124.44637307, ..., -127.12167483,
           -127.12167483, -127.12167483])

  • depth (m)
    (index)
    float64
    0.9907 0.9497 ... 0.5305 0.1026

    array([0.9907086 , 0.9496902 , 0.8640232 , ..., 1.1711848 , 0.53051794,
           0.1025867 ])

  • cdom (ppb/L)
    (index)
    float64
    1.348 1.348 1.348 ... 1.798 1.798

    array([1.3485, 1.3485, 1.3485, ..., 1.7081, 1.798 , 1.798 ])

  • cdom_qc
    (index)
    float64
    0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0

    array([0., 0., 0., ..., 0., 0., 0.])

  • conductivity (S m-1)
    (index)
    float64
    nan nan nan nan ... 3.759 nan nan

    array([      nan,       nan,       nan, ..., 3.7591789,       nan,
                 nan])

  • conductivity_qc
    (index)
    float64
    4.0 4.0 4.0 4.0 ... 1.0 1.0 4.0 4.0

    array([4., 4., 4., ..., 1., 4., 4.])

  • density (kg m-3)
    (index)
    float64
    nan nan nan ... 1.025e+03 nan nan

    array([      nan,       nan,       nan, ..., 1024.5771,       nan,
                 nan])

  • density_qc
    (index)
    float64
    1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0

    array([1., 1., 1., ..., 1., 1., 1.])

  • depth_qc
    (index)
    float64
    0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0

    array([0., 0., 0., ..., 0., 0., 0.])

  • fluorescence (microgram/L)
    (index)
    float64
    0.4148 0.3416 ... 0.549 0.5246

    array([0.4148, 0.3416, 0.3538, ..., 0.5368, 0.549 , 0.5246])

  • fluorescence_qc
    (index)
    float64
    0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0

    array([0., 0., 0., ..., 0., 0., 0.])

  • instrument_ctd (1)
    (index)
    float64
    nan nan nan nan ... nan nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • lat_qc
    (index)
    float64
    0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0

    array([0., 0., 0., ..., 0., 0., 0.])

  • lat_uv (degrees_north)
    (index)
    float64
    40.83 40.83 40.83 ... 41.05 41.05

    array([40.83496667, 40.83496667, 40.83496667, ..., 41.05235689,
           41.05235689, 41.05235689])

  • lat_uv_qc
    (index)
    float64
    1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0

    array([1., 1., 1., ..., 1., 1., 1.])

  • latitude_qc
    (index)
    float64
    1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0

    array([1., 1., 1., ..., 1., 1., 1.])

  • lon_qc
    (index)
    float64
    0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0

    array([0., 0., 0., ..., 0., 0., 0.])

  • lon_uv (degrees_east)
    (index)
    float64
    -124.4 -124.4 ... -127.1 -127.1

    array([-124.44637307, -124.44637307, -124.44637307, ..., -127.12167483,
           -127.12167483, -127.12167483])

  • lon_uv_qc
    (index)
    float64
    1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0

    array([1., 1., 1., ..., 1., 1., 1.])

  • longitude_qc
    (index)
    float64
    1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0

    array([1., 1., 1., ..., 1., 1., 1.])

  • opbs (m^-1)
    (index)
    float64
    0.001734 0.001614 ... 0.0002276

    array([0.0017343 , 0.00161437, 0.00163897, ..., 0.0001968 , 0.00020603,
           0.00022755])

  • opbs_qc
    (index)
    float64
    0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0

    array([0., 0., 0., ..., 0., 0., 0.])

  • oxygen (micromol/kg)
    (index)
    float64
    nan nan nan nan ... nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • oxygen_qc
    (index)
    float64
    4.0 4.0 4.0 4.0 ... 1.0 3.0 4.0 4.0

    array([4., 4., 4., ..., 3., 4., 4.])

  • platform_meta (1)
    (index)
    float64
    nan nan nan nan ... nan nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • precise_lat (degrees_north)
    (index)
    float64
    40.83 40.83 40.83 ... 41.05 41.05

    array([40.83468333, 40.83468327, 40.83468315, ..., 41.05268175,
           41.05268137, 41.05268001])

  • precise_lat_qc
    (index)
    float64
    nan nan nan nan ... nan nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • precise_lon (degrees_east)
    (index)
    float64
    -124.4 -124.4 ... -127.1 -127.1

    array([-124.4462    , -124.44620185, -124.44620555, ..., -127.12957832,
           -127.12958763, -127.12959082])

  • precise_lon_qc
    (index)
    float64
    nan nan nan nan ... nan nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • precise_time (UTC)
    (index)
    object
    '2023-01-26T20:06:39Z' ... '2023...

    array(['2023-01-26T20:06:39Z', '2023-01-26T20:06:44Z',
           '2023-01-26T20:06:49Z', ..., '2023-05-23T12:34:17Z',
           '2023-05-23T12:34:22Z', '2023-05-23T12:34:27Z'], dtype=object)

  • precise_time_qc
    (index)
    float64
    1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0

    array([1., 1., 1., ..., 1., 1., 1.])

  • pressure (dbar)
    (index)
    float64
    0.9992 0.9578 ... 0.5347 0.1037

    array([0.99918216, 0.9578449 , 0.87150395, ..., 1.1801987 , 0.53466433,
           0.10365458])

  • pressure_qc
    (index)
    float64
    0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0

    array([0., 0., 0., ..., 0., 0., 0.])

  • salinity (0.001)
    (index)
    float64
    nan nan nan nan ... nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • salinity_qc
    (index)
    float64
    4.0 4.0 4.0 4.0 ... 1.0 3.0 4.0 4.0

    array([4., 4., 4., ..., 3., 4., 4.])

  • temperature (Celsius)
    (index)
    float64
    11.15 11.15 11.15 ... 12.23 12.19

    array([11.150965 , 11.150085 , 11.149957 , ..., 12.228412 , 12.2253275,
           12.190469 ])

  • temperature_qc
    (index)
    float64
    1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0

    array([1., 1., 1., ..., 1., 1., 1.])

  • time_qc
    (index)
    float64
    1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0

    array([1., 1., 1., ..., 1., 1., 1.])

  • time_uv (UTC)
    (index)
    object
    '2023-01-26T20:10:37Z' ... '2023...

    array(['2023-01-26T20:10:37Z', '2023-01-26T20:10:37Z',
           '2023-01-26T20:10:37Z', ..., '2023-05-23T10:49:13Z',
           '2023-05-23T10:49:13Z', '2023-05-23T10:49:13Z'], dtype=object)

  • time_uv_qc
    (index)
    float64
    1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0

    array([1., 1., 1., ..., 1., 1., 1.])

  • u (m s-1)
    (index)
    float64
    -0.06249 -0.06249 ... nan nan

    array([-0.06249343, -0.06249343, -0.06249343, ...,         nan,
                   nan,         nan])

  • u_qc
    (index)
    float64
    3.0 3.0 3.0 3.0 ... nan nan nan nan

    array([ 3.,  3.,  3., ..., nan, nan, nan])

  • v (m s-1)
    (index)
    float64
    -0.00278 -0.00278 ... nan nan

    array([-0.0027802, -0.0027802, -0.0027802, ...,        nan,        nan,
                  nan])

  • v_qc
    (index)
    float64
    3.0 3.0 3.0 3.0 ... nan nan nan nan

    array([ 3.,  3.,  3., ..., nan, nan, nan])

• Indexes: (1)
  • index
    PandasIndex

    PandasIndex(Int64Index([     0,      1,      2,      3,      4,      5,      6,      7,
                     8,      9,
                ...
                820109, 820110, 820111, 820112, 820113, 820114, 820115, 820116,
                820117, 820118],
               dtype='int64', name='index', length=820119))

• Attributes: (0)

This imidiately decrease the table from 90984 rows to 39661 rows.

Creating 3D plotting function

To take a quick look on the glider vertical profile, we construct a plotting function using the matplotlib package which could avoid rewriting the plotting code over and over.

import matplotlib.pyplot as plt

def plot_3d_view(ele_angle=-140,hori_angle=60):
    """
    This function is designed for creating axe object) for plotting the 
    3D scatter plot.

    Parameters
    ----------
    ele_angle : integer
        elevation angle of viewing the 3D scatter plot
    hori_angle : integer
        horizontal angle of viewing the 3D scatter plot
    Returns
    -------
    ax : matplotlib axe object
        the axe object that can apply scatter3D method 

    Raises
    ------
    """
    fig = plt.figure(figsize=(5,5))
    ax = plt.axes([0,0,1.5,1],projection = '3d')
    ax.view_init(ele_angle, hori_angle)
    ax.set_xlabel('Longitude')
    ax.set_ylabel('Latitude')
    ax.set_zlabel('Depth')
    return ax

Plotting a subset of the dataset and focusing on single variables

• temperature profile along the glider path

ds_part = ds_transect
varname = 'temperature (Celsius)'
zname = 'depth (m)'
yname = 'latitude (degrees_north)'
xname = 'longitude (degrees_east)'

ax2 = plot_3d_view(ele_angle=-140,hori_angle=60)
p = ax2.scatter3D(ds_part[xname],
                  ds_part[yname],
                  ds_part[zname],
                  c=ds_part[varname],      # color value of individual points is taken from their heights
                  cmap="viridis"                           # the color mapping to be used. Other example options: winter, autumn, ...
                )
# ax2.invert_zaxis()
ax2.invert_xaxis()
cbar = plt.colorbar(p)
cbar.set_label(varname)

• salinity profile along the glider path

varname = 'salinity (0.001)'
zname = 'depth (m)'
yname = 'latitude (degrees_north)'
xname = 'longitude (degrees_east)'

ax2 = plot_3d_view(ele_angle=-160,hori_angle=110)
p = ax2.scatter3D(ds_part[xname],
                  ds_part[yname],
                  ds_part[zname],
                  c=ds_part[varname],      # color value of individual points is taken from their heights
                  cmap="viridis"                           # the color mapping to be used. Other example options: winter, autumn, ...
                )
# ax2.invert_zaxis()
ax2.invert_xaxis()
cbar = plt.colorbar(p)
cbar.set_label(varname)

Showing the geopgraphic location of the glider path

Using the FOLIUM package, we can plot a interactive map in the notebook environment to investigate the path of the glider.

import folium
import numpy as np

lon = ds_part[xname].data
lat = ds_part[yname].data

fmap = folium.Map(location=[(np.min(lat)+np.max(lat))/2, (np.min(lon)+np.max(lon))/2], tiles="OpenStreetMap", zoom_start=8)
points = [[lat[i],lon[i]] for i in range(len(lon)) ]
folium.PolyLine(points, color='red', weight=2.5, opacity=0.4,popup=f'{dataset_id}').add_to(fmap)
folium.Marker([lat[0],lon[0]], popup=f'start').add_to(fmap)
folium.Marker([lat[-1],lon[-1]], popup=f'end').add_to(fmap)
fmap

Make this Notebook Trusted to load map: File -> Trust Notebook

Converting the table data to gridded data

In the following steps, we demostrate how to convert a point observation from glider to a gridded data. The gridded dataset will have two dimensions - time as first dimension and depth as the second dimension.

By using the numpy.unique method, we first establish the two dimensions and the assoicated arrays.

import pandas as pd

# idealy/theoretically the three array should have same len (first dimension)
# lat = np.unique(ds_transect['latitude (degrees_north)'].data)
# lon = np.unique(ds_transect['longitude (degrees_east)'].data)
time = np.unique(ds_transect['time (UTC)'].data)
dtime = pd.to_datetime(time)

# vertical profile (second dimension)
depth = np.unique(ds_transect['depth (m)'].data)

Then, we create an empty xr.DataArray to later put the interpolated values into the corresponding dimensions.

# create 2D gridded array for vertical profile along the glider trajectory
da_transect = xr.DataArray(
    coords={
        "depth": depth,
        "time": dtime.values,
        # "lon": ("time", lon),  # place holder not real lon at the associated time
        # "lat": ("time", lat),  # place holder not real lat at the associated time
    },
    dims=["depth", "time"],
)
da_transect = da_transect.rename('var')

To convert the table data into gridded data, we perform the following processes

1. for each single time profiling, we take all the vertical measurement

2. for each single time profiling, we average the measurement on the same vertical level

3. if multiple location is registered during the vertical profiling, we use the first recorded location as the profiling location

4. the veritical profile is linearly interpolated to make every vertical profiling to have the same number of grid

# we minimized the number of time profiling (first 10 time stamp) that needed to be processed to gridded data in this notebook (efficiency)
print(f'processing {varname}')
for i in range(len(time[:10])):
    print(time[i])
    loc_time = time[i]
    
    # only preserve single time (mid point time during the profiling)
    ds_temp = ds_transect.where((ds['time (UTC)'] == loc_time),drop=True)
    ds_temp = ds_temp.assign_coords({'depth':ds_temp['depth (m)']})
    
    # group mean the value for up and down profiling (single depth single value)
    ds_temp = ds_temp.groupby('depth (m)').mean()
    
    # create profile data array to store profiling at one location
    da_var = xr.DataArray(
        ds_temp[varname].data,
        coords={"depth": ds_temp['depth (m)'].data},
        dims=["depth"]
    )
    da_var = da_var.rename('var')

    # put the single profiling to the 2D gridded dataarray
    da_var = da_var.interp(
        depth=depth,
        method="linear",
        kwargs={"fill_value": np.nan},
    )
    da_transect[:,i] = da_var 

processing salinity (0.001)
2023-01-26T20:10:27Z

2023-01-26T20:14:27Z

2023-01-26T20:35:57Z

2023-01-26T20:40:19Z

2023-01-26T21:05:05Z

2023-01-26T21:33:42Z

2023-01-26T22:08:21Z

2023-01-26T22:40:14Z

2023-01-26T23:27:47Z

2023-01-27T00:17:52Z

The gridded dataset that has the interpolated values put in is

da_transect

<xarray.DataArray 'var' (depth: 809648, time: 996)>
array([[nan, nan, nan, ..., nan, nan, nan],
       [nan, nan, nan, ..., nan, nan, nan],
       [nan, nan, nan, ..., nan, nan, nan],
       ...,
       [nan, nan, nan, ..., nan, nan, nan],
       [nan, nan, nan, ..., nan, nan, nan],
       [nan, nan, nan, ..., nan, nan, nan]])
Coordinates:
  * depth    (depth) float64 -0.3087 -0.3073 -0.3015 ... 993.3 993.4 993.4
  * time     (time) datetime64[ns] 2023-01-26T20:10:27 ... 2023-05-23T10:49:13

xarray.DataArray

'var'

• depth: 809648
• time: 996

• nan nan nan nan nan nan nan nan ... nan nan nan nan nan nan nan nan

  array([[nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         ...,
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan],
         [nan, nan, nan, ..., nan, nan, nan]])

• Coordinates: (2)
  • depth
    (depth)
    float64
    -0.3087 -0.3073 ... 993.4 993.4

    array([-3.087025e-01, -3.072967e-01, -3.014633e-01, ...,  9.933227e+02,
            9.933670e+02,  9.933872e+02])

  • time
    (time)
    datetime64[ns]
    2023-01-26T20:10:27 ... 2023-05-...

    array(['2023-01-26T20:10:27.000000000', '2023-01-26T20:14:27.000000000',
           '2023-01-26T20:35:57.000000000', ..., '2023-05-23T05:06:28.000000000',
           '2023-05-23T08:04:02.000000000', '2023-05-23T10:49:13.000000000'],
          dtype='datetime64[ns]')

• Indexes: (2)
  • depth
    PandasIndex

    PandasIndex(Float64Index([-0.30870247, -0.30729666, -0.30146325, -0.28009477, -0.27651685,
                  -0.27498317, -0.26061705, -0.25455007, -0.25251982, -0.24778113,
                  ...
                    992.88434,    993.1243,   993.12775,    993.2039,   993.23956,
                    993.25336,   993.30536,    993.3227,     993.367,    993.3872],
                 dtype='float64', name='depth', length=809648))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2023-01-26 20:10:27', '2023-01-26 20:14:27',
                   '2023-01-26 20:35:57', '2023-01-26 20:40:19',
                   '2023-01-26 21:05:05', '2023-01-26 21:33:42',
                   '2023-01-26 22:08:21', '2023-01-26 22:40:14',
                   '2023-01-26 23:27:47', '2023-01-27 00:17:52',
                   ...
                   '2023-05-22 08:46:18', '2023-05-22 11:44:01',
                   '2023-05-22 14:52:12', '2023-05-22 17:49:56',
                   '2023-05-22 20:53:39', '2023-05-22 23:40:53',
                   '2023-05-23 02:19:58', '2023-05-23 05:06:28',
                   '2023-05-23 08:04:02', '2023-05-23 10:49:13'],
                  dtype='datetime64[ns]', name='time', length=996, freq=None))

• Attributes: (0)

Plotting the gridded data

da_transect.where(da_transect.notnull(),drop=True).plot(yincrease=False, x='time', y='depth')

<matplotlib.collections.QuadMesh at 0x7f8784aafb50>